Collaborative Construction of Multimodal Utterances

The papers in this volume demonstrate the pervasiveness of multimodal utterances. The collaborative construction of utterances is also well known. In this chapter we explore utterances that are both multimodal and collaboratively constructed; in particular, utterances in which the gesture of one participant stands in a relation of mutual elaboration with the talk of another participant. Drawing on interactions between Japanese airline pilots and an American flight instructor, we show how when multiple participants talk and gesture simultaneously, relations among semiotic resources proliferate. Pilots use their speech and their bodies to jointly construct multimodal representations of the objects, the events, and the actions that constitute their profession. 1 Department of Cognitive Science, University of California San Diego 2 Department of Communication, Cornell University, Ithaca, NY